Patent Document 1 (Japanese Patent Laid-Open Publication No. 8-335136) discloses a pointing device comprising image capturing means. This pointing device is used so as to designate an arbitrary position on a display screen. The pointing device detects two markers in an image captured by the image capturing means, and calculates positions of the markers in the image. Thereafter, a position on the display screen which is designated by the pointing device (designated position) is calculated from the positions of the markers. Patent Document 1 also describes that, when makers having two colors (red and green) are used, the colors of the markers are not displayed on the display screen (paragraph 0029), and when the two markers are not detected in a captured image, an error process is performed (paragraph 0042).
As described above, Patent Document 1 describes that, when the two markers are detected, a position designated by the pointing device is calculated. However, in the technique of Patent Document 1, when the two markers are not correctly detected due to an influence of noise, the designated position cannot be correctly calculated. Examples of the noise include an image which is displayed on the display screen and has the same color as that of the marker, an object which is present outside the display screen and has the same color as that of the marker, and the like. In other words, when an image having the same color as that of the marker is displayed on the screen or when an object having the same color as that of the marker is present outside the display screen, three or more images are detected as markers, so that the two markers cannot be correctly detected, and therefore, the designated position is not likely to be correctly calculated.
As described above, in the technique of Patent Document 1, only a situation that the two markers can be captured and identified is assumed. Therefore, the technique of Patent Document 1 cannot address a situation that something other than the markers, such as an image on the display screen, an object outside the display screen, or the like, is detected as the marker. Therefore, the technique of Patent Document 1 cannot remove the influence of noise.
Therefore, an object of the exemplary illustrative non-limiting implementation is to provide a coordinate calculating apparatus and a coordinate calculating program which can remove the influence of noise.
The exemplary illustrative non-limiting implementation has the following features to attain the object mentioned above. Note that, in this section, reference numerals, supplemental descriptions, and the like in parentheses indicate correspondence with non-limiting implementations described below so as to assist understanding of the exemplary illustrative non-limiting implementation, but do not limit the exemplary illustrative non-limiting implementation.
An exemplary illustrative non-limiting implementation is directed to a coordinate calculating apparatus (a game apparatus 3 and a controller 7) for calculating two object position coordinate points (marker coordinate points) from an image captured by an image capturing apparatus (an image capturing element 40), the two object position coordinate points indicating positions of two predetermined objects (markers 8a and 8b) whose images to be captured, the positions being on a plane (a coordinate plane illustrated in FIG. 9) corresponding to the captured image. The apparatus comprises coordinate detecting means (an image processing circuit 41), reference setting means (a CPU 10 executing step S35 or step S75, etc.; hereinafter only step numbers are described), and selection means (S81). The coordinate detecting means detects a position coordinate point (a detected coordinate point) of an area satisfying a predetermined condition in the captured image for each area. The reference setting means sets a reference (a previous vector or an assumed vector) with respect to at least one of a direction connecting two position coordinate points in the captured image and a length between the two position coordinate points. The selection means, when a plurality of position coordinate points are detected by the coordinate detecting means, with respect to combinations of two position coordinate points among the plurality of position coordinate points, compares at least one of a direction connecting two position coordinate points in the combinations and a length between the two position coordinate points, with the reference, and selects a combination of two position coordinate points in the combinations most close to the reference as the two object position coordinate points.
In a further exemplary illustrative non-limiting implementation, the reference setting means calculates a first vector (an assumed vector) indicating a direction on the plane, based on a tilt detected by tilt determining means (preferably including a linear acceleration sensor 37) which determines a tilt of the image capturing apparatus, where the direction connecting the two position coordinate points is used as a reference. The selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector.
In another exemplary illustrative non-limiting implementation, the selection means may calculate a similarity from an inner product value of a vector connecting two position coordinate points among the plurality of position coordinate points and the first vector.
In an additional exemplary illustrative non-limiting implementation, the reference setting means may calculate the first vector based on a correspondence relationship between a tilt determined by the tilt determining means in a situation that the image capturing apparatus is at a predetermined tilt, and a direction connecting the two object position coordinate points detected from an image captured by the image capturing apparatus in the situation.
In yet another exemplary illustrative non-limiting implementation, the coordinate calculating apparatus may further comprise object position storing means (a main memory 13) of storing object position coordinate data (marker coordinate data 534a and 534b) including at least one of a set of the two object position coordinate points selected by the selection means, and a vector indicating a direction connecting the two object position coordinate points, every time selection is performed by the selection means. In this case, the coordinate detecting means sequentially detects position coordinate points from images sequentially captured by the image capturing apparatus. The reference setting means calculates or obtains a second vector (a previous vector) which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored into the object position storing means, every time the coordinate detecting means detects a position coordinate point, and sets the second vector as the reference. The selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector.
In another exemplary illustrative non-limiting implementation, the selection means may determine a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the vector and a similarity between a length between two position coordinate points and a length of the second vector.
In another exemplary illustrative non-limiting implementation, the coordinate calculating apparatus may further comprise first determination means (step S11) of determining whether or not the object position coordinate data has been stored into the object position storing means by selections performed from a predetermined number of times before until the previous time, before the reference setting means sets the reference. In this case, the reference setting means calculates or obtains the second vector from most recently stored object position coordinate data only when the result of the determination by the first determination means is positive.
In a further exemplary illustrative non-limiting implementation, the selection means may calculate a similarity from an inner product value of a vector connecting two position coordinate points from the plurality of position coordinate points, and the second vector.
In a further exemplary illustrative non-limiting implementation, the coordinate calculating apparatus may further comprise object position storing means and second determination means (step S11). The object position storing means stores object position coordinate data including at least one of a set of the two object position coordinate points selected by the selection means, and a vector indicating a direction connecting the two object position coordinate points, every time selection is performed by the selection means. The second determination means determines whether or not two object position coordinate points have been selected at least once among selections performed by the selection means from a predetermined number of times before. The coordinate detecting means sequentially detects position coordinate points from images sequentially captured by the image capturing apparatus. The reference setting means sets a first vector indicating a direction on the plane based on a tilt determined by tilt determining means for a tilt of the image capturing apparatus, using a direction connecting two position coordinate points on the plane as a reference, when a result of the determination by the second determination means is positive; and calculates or obtains a second vector which is a vector indicating a direction connecting two object position coordinate points from object position coordinate data most recently stored by the object position storing means when the result of the determination by the second determination means is negative. The selection means determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the first vector when the result of the determination by the second determination means is positive; and determines a combination of two position coordinate points to be selected, based on a similarity between a direction connecting two position coordinate points and the direction of the second vector when the result of the determination by the second determination means is negative.
In a still further exemplary illustrative non-limiting implementation, the tilt determining means may include an acceleration sensor (37) which is fixedly provided to a housing used in common with the image capturing apparatus, and detects accelerations in two axial directions perpendicular to an image capturing direction of the image capturing apparatus. In this case, the reference setting means calculates the first vector based on the accelerations in the axial directions detected by the acceleration sensor.
Note that the exemplary illustrative non-limiting implementation may be provided in the form of a coordinate calculating program which is executed by a computer of a coordinate calculating apparatus.
In an exemplary illustrative non-limiting implementation, by using a reference set by the reference setting means, two object position coordinate points can be selected from a plurality of position coordinate points. Therefore, even when three or more position coordinate points are detected by the coordinate detecting means due to an influence of noise, two position coordinate points can be specified among them, and thus, the influence of noise can be removed. Thereby, position coordinate points of objects whose images to be captured can be calculated even in the presence of an influence of noise.
In another exemplary illustrative non-limiting implementation, the tilt determining means is used to determine a tilt of the image capturing apparatus, and a first vector corresponding to the tilt is used as a reference, thereby making it possible to select two object position coordinate points, taking into consideration the tilt of the image capturing apparatus. A direction connecting the two object position coordinate points corresponds to the tilt of the image capturing apparatus. Therefore, by selecting two object position coordinate points while taking into consideration the tilt of the image capturing apparatus, the two object position coordinate points can be correctly selected.
According to the third aspect, by calculating an inner product value of vectors, a similarity between the vectors can be correctly calculated.
According to the fourth aspect, the first vector is calculated based on a correspondence relationship between the tilt of the image capturing apparatus and the direction connecting the object position coordinate points, thereby making it possible to correctly calculating the first vector corresponding to the tilt of the image capturing apparatus.
According to the fifth aspect, two object position coordinate points are selected using a second vector indicating a positional relationship between previous object position coordinate points. Here, when the selection process is executed at a frequency of several ten times or several hundred times per second, it can be assumed that the amount of a change from a previous object position coordinate point to a current object position coordinate point is not large. Therefore, by selecting two object position coordinate points using the second vector calculated based on the previous object position coordinate as a reference, the two object position coordinate points can be correctly selected.
According to the sixth aspect, a similarity between vectors is calculated, taking into consideration the lengths of a vector connecting two position coordinate points and the second vector, thereby making it possible to more correctly select two object position coordinate points.
According to the seventh aspect, only when most recently stored object position coordinate data is data stored after a predetermined number of times before, the second vector is calculated from the object position coordinate data. Here, when the most recently stored object position coordinate data is excessively old, the second vector obtained from the object position coordinate data and a vector obtained from current object position coordinate points are no longer very similar to each other. In this case, if the second vector is used as a reference to select position coordinate points, correct object position coordinate points can be no longer selected. In contrast to this, according to the seventh aspect, excessively old object position coordinate data is not used to calculate the second vector, whereby an incorrect object position coordinate point can be prevented from being calculated.
According to the eighth aspect, a similarity in direction between vectors can be correctly calculated by calculating an inner product value of the vectors.
According to the ninth aspect, when the number of position coordinate points to be selected is one or less for each selection in the selection means from a predetermined number of times before until the previous time, the second vector is not used as a reference, and the first vector is used as a reference. This is because, in the above-described case, the possibility that the second vector calculated from previous object position coordinate data has correct object position coordinate points is low. In other words, according to the ninth aspect, only when the possibility that correct object position coordinate points are obtained is high, the previous object position coordinate data is used, and if not, data of a tilt determined by the tilt determining means is used without using the previous object position coordinate data. Thereby, a method of using previous object position coordinate data or a method of using a tilt determine by the tilt determining means can be appropriately selected and used, depending on the situation.
According to the tenth aspect, by using an acceleration sensor, a tilt of the image capturing apparatus can be easily determined by calculating a tilt angle from the linear acceleration signal output by the acceleration sensor 73 in response to the acceleration of gravity (i.e., 9.8 m/S.sup.2).